Skip to content

Resolve relative inline links to fully-qualified URLs during scrape - #129

Open
arimu1 wants to merge 2 commits into
timescale:mainfrom
arimu1:fix/12-fully-qualified-scrape-links
Open

Resolve relative inline links to fully-qualified URLs during scrape#129
arimu1 wants to merge 2 commits into
timescale:mainfrom
arimu1:fix/12-fully-qualified-scrape-links

Conversation

@arimu1

@arimu1 arimu1 commented Jul 21, 2026

Copy link
Copy Markdown

Summary

Closes #12

When scraping documentation pages, relative <a href> links in the source HTML (e.g. /self-hosted/latest/migration/entire-database/) were carried verbatim into the generated markdown. Per the issue, this produces broken-looking inline links when an LLM quotes the returned markdown chunks, since the link target isn't resolvable without the source page's base URL.

This resolves relative links against the HTML source (per the issue's suggestion), before the markdown conversion step, across all three ingest pipelines:

  • ingest/tiger_docs.py — new resolve_relative_links method on SitemapMarkdownSpider, called in parse() with the current page's response.url as the base, right after the existing strip_data_images step.
  • ingest/postgis_docs.py — calls the new shared resolve_relative_links util with each page's resolved full_url as the base.
  • ingest/postgres_docs.py — calls the same shared util with a per-page URL built from POSTGRES_BASE_URL, the version, and the page's slug (threaded version into build_markdown, which didn't have it before).
  • ingest/utils/beautiful_soup.py — adds the shared resolve_relative_links(soup, base_url) helper (urljoin was already imported here but unused).

Already-absolute links, mailto:/tel: links, etc. are left untouched by urljoin.

Note: this also expands #fragment-only anchors (same-page links) to fully-qualified URLs, since urljoin treats them like any other relative reference. The targets are correct, and it's arguably desirable in a standalone markdown corpus, but PostgreSQL/PostGIS docs are anchor-heavy, so it will visibly increase diff volume in the regenerated output — flagging so it's not a surprise.

Test plan

No Python test harness exists in this repo (ingest/ has no test files and CI (.github/workflows/test.yml) only runs bun test), so I verified behavior manually against the exact example from the issue:

$ cd ingest && PYTHONPATH=.. uv run python - <<'PY'
from bs4 import BeautifulSoup
from markdownify import markdownify
from ingest.utils.beautiful_soup import resolve_relative_links

html = '<main><a href="/self-hosted/latest/migration/entire-database/">migrate your entire database at once</a></main>'
soup = BeautifulSoup(html, "html.parser")
soup = resolve_relative_links(soup, "https://docs.tigerdata.com/self-hosted/latest/migration/some-page/")
print(markdownify(str(soup), heading_style="ATX").strip())
PY
[migrate your entire database at once](https://docs.tigerdata.com/self-hosted/latest/migration/entire-database/)
  • Confirmed relative links resolve to fully-qualified URLs, matching the issue's before/after example
  • Confirmed already-absolute links and same-page fragment links are handled correctly by urljoin
  • uvx ruff check . / uvx ruff format --check . show no new issues introduced by this change (diffed against main, all remaining warnings pre-exist on main)
  • python -m ast parses all four changed files without error

🤖 Generated with Claude Code

arimu1 added 2 commits July 21, 2026 09:22
Relative hrefs from source HTML (e.g. /self-hosted/latest/migration/entire-database/)
were being carried verbatim into the markdown, producing links that
LLMs can't resolve on their own. Rewrite <a href> links against each
page's URL before converting to markdown, across the Tiger, PostGIS,
and Postgres ingest pipelines.

Closes timescale#12
The spider method had its own divergent implementation of link
resolution (conditional rewrite + debug counter) alongside the shared
ingest.utils.beautiful_soup.resolve_relative_links (unconditional
rewrite) used by the other two pipelines. Same name, same job, two
bodies that could silently drift. The spider method is now a thin
wrapper that delegates to the shared util and keeps its
debug-level resolved-link count.
@arimu1

arimu1 commented Jul 21, 2026

Copy link
Copy Markdown
Author

Addressed the review feedback:

  • Duplicate implementation (blocking): SitemapMarkdownSpider.resolve_relative_links in ingest/tiger_docs.py had its own divergent body (conditional rewrite + debug counter) alongside the shared ingest.utils.beautiful_soup.resolve_relative_links used by the other two pipelines. It's now a thin wrapper that delegates to the shared util (imported from ingest.utils.beautiful_soup) and keeps its debug-level "Resolved N relative links" logging. parse() still calls it with response.url as base, unchanged (ingest/tiger_docs.py:966). Manually verified the wrapper against a mix of relative, fragment-only, and already-absolute hrefs — correct resolution and correct count logged.
  • PR body: added a note that #fragment-only anchors also get expanded to fully-qualified URLs (since urljoin treats them like any other relative reference) — correct, but will visibly increase diff volume for the anchor-heavy Postgres/PostGIS docs, so it shouldn't be a surprise.
  • No tests added, per the earlier confirmation that this repo has no Python test infra (CI only runs bun test) — verified manually as before.

uvx ruff check . / uvx ruff format --check . show no new issues (same 9 pre-existing warnings as on main, none touching the changed lines).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make sure inline links are fully qualified URLs during scrape process

1 participant